Changing an application state using neurological data

ABSTRACT

Computer systems, methods, and storage media for changing the state of an application by detecting neurological user intent data associated with a particular operation of a particular application state, and changing the application state so as to enable execution of the particular operation as intended by the user. The application state is automatically changed to align with the intended operation, as determined by received neurological user intent data, so that the intended operation is performed. Some embodiments relate to a computer system creating or updating a state machine, through a training process, to change the state of an application according to detected neurological data.

BACKGROUND

Neurological data can be gathered through a variety of techniques. Onenon-invasive technique is electroencephalography (EEG), which involvesthe placement of electrodes along the scalp of a user or subject tomeasure voltage fluctuations resulting from ionic current within theneurons of the brain. EEG is often used in clinical contexts to monitorsleep patterns or to diagnose epilepsy.

Some computer applications include various application states, where thesame user input is operable to cause different functions within theapplication depending on the particular application state that theapplication is in when the user input is received. By way of example, acombination of ‘click and drag’ input is operable to perform a varietyof different functions within some 3D modeling software, depending onthe state of the application at the time the input is received (e.g.,selecting and moving an object, rotating the object, drawing a line onthe object, defining an edge on the application, extruding a surfacefrom the object, and so forth.

When a user desires to perform a particular operation that is notassociated with the current application state, the user must manuallychange from the current application state to another application stateby making a specific menu selection or by otherwise providing input thatis operable for changing the application to the desired applicationstate.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments describedabove. Rather, this background is only provided to illustrate exemplarytechnology areas where some embodiments described herein may bepracticed.

BRIEF SUMMARY

The present disclosure relates to computer systems, methods, andcomputer storage media configured for facilitating application statechanges. In some embodiments, the state changes are performed withdetected neurological user intention data associated with a particularoperation of a particular application state, and changing theapplication state so as to enable execution of the particular operationas intended by the user and without requiring a user to manually changethe application state. In some embodiments, the application state isautomatically changed to align with the intended operation, asdetermined by received neurological user intent data.

Some embodiments include a computer system operating an application thatis configured to change state and that performs a first operation inresponse to detecting a particular gesture when in a first state, and asecond operation in response to detecting the particular gesture when ina second state. When in the first state, the computer system detectsneurological user intention data associated with the second operation.In response, the computing system changes the application from the firststate to the second state, enabling execution of the intended secondoperation and without requiring user input to manually change theapplication state.

Some embodiments relate to a computer system generating or updating astate machine configured to change the state of an application accordingto detected neurological data. The computer system associates a set ofdifferent application operations to a corresponding set of applicationstates. A particular gesture causes the application to perform aparticular operation of the set of operation based on a correspondingapplication state. The computer system detects first neurological userintention data generated when the user is performing the particulargesture to actuate a first operation, and detects second neurologicaluser intention data generated when the user is performing the particulargesture to actuate a second operation. The computer system subsequentlymaps or otherwise associates the first and second neurological userintention data with the first and second operations, respectively,within an appropriate stored data structure.

Some embodiments relate to a computer system generating or updating astate machine by detecting first neurological user intention datagenerated by a user when the user changes the application to aparticular state in order to perform a desired particular operation, anddetecting second neurological user intention data generated by the userduring performance of the gesture that actuates the particularoperation. The computer system then maps or otherwise associates thefirst neurological user intention data with the particular applicationstate and the second neurological user intention data with theparticular operation within an appropriate stored data structure.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates a computer system that can be used to change thestate of an application using neurological user intent data;

FIG. 2 illustrates a flowchart of an exemplary method for changing thestate of an application using neurological user intent data;

FIG. 3 illustrates a flowchart of an exemplary method for generating ormanaging a state machine;

FIG. 4 illustrates a flowchart of an exemplary method for generating ormanaging a state machine;

FIGS. 5A-5D illustrate operation of a state machine to automaticallychange the state of an exemplary application to enable execution of anintended application operation, as determined using neurological userintent data; and

FIG. 6 illustrates operation of a state machine to expressly change thestate of an exemplary application using neurological user intent data.

DETAILED DESCRIPTION

The present disclosure relates to computer systems, methods, andcomputer storage media for changing a state of an application bydetecting neurological user intention data associated with a particularoperation of a particular application state, and changing theapplication state so as to enable execution of the particular operationas intended by the user.

In some embodiments, the neurological user intention data is associatedwith a particular gesture that causes different application operationsamong a set of application operations, depending on the current state ofthe application. In some embodiments, the application state isautomatically changed in response to received neurological userintention data, and the particular operation associated with theneurological user intention data is automatically performed, in responseto receiving the neurological user intention data.

Various technical effects and benefits may be achieved by implementingaspects of the disclosed embodiments. For instance, some disclosedembodiments are operable to facilitate efficient selection and/or changeof application state for computer applications having multiple states.This can be particularly beneficial for applications in which the sameor similar gestures will selectively cause the performance of differentoperations depending on the different application state settings. Theuse of neurological user intent data can help facilitate user selectionof different states and/or functions by at least reducing the number ofmanual inputs that have to be entered and processed for changing states.

By way of example, at least some of the disclosed embodiments enableefficient use of computer applications having multiple states,particularly where the same or similar gestures cause the performance ofdifferent operations according to the particular state setting of theapplication. For example, where an application includes multiple modesof action for a single gesture depending on the particular state settingof the application, a user of a prior computer system is required tomanually switch between state settings to perform different desiredoperations, often through cumbersome menu navigation steps, even thoughthe operations are actuated through the same user gesture. This prolongsthe time taken to perform desired tasks using the application, as theuser is required to access a menu option or use some other manualselection process to switch between multiple states to allow thesequence of desired operations to be performed. In contrast, thedisclosed embodiments provide systems that are more efficient andprovide increase user convenience.

Further, in at least some circumstances, mistakes are made as a userperforms a gesture with the intent to cause performance of a specificoperation, without realizing that the application is presently in astate that associates a different mode of action to the performedgesture. The user will have thereby inadvertently caused performance ofan undesired operation. Such missteps require additional operationalsteps to undo, mitigate, or otherwise fix the operational error, addingto computer system processing requirements and processing time, inaddition to adding to user frustration.

In contrast, one or more of the disclosed embodiments enable automaticchange to the intended application state, so that the intended operationis performed, even without requiring the user to manually change theapplication state. The disclosed embodiments are therefore usable toimprove the overall user experience with applications having variousdifferent states and which are responsive to one or more gesturesaccording to different modes of operation. The error reduction canreduce user frustration as well as reduce the amount of computerprocessing involved to achieve a given set of operations. Further, evenin circumstances where a user is careful and errors are minimal, theembodiments disclosed herein are usable to reduce the time and overallnumber of steps required to achieve a desired result by reducing oreliminating the need to manually switch between application statesbetween different desired operations.

As used herein, the term “gesture” includes movements and motions madeby one or more body parts of a user. In some embodiments, a gesture neednot be actually physically performed by a user to give functionaleffect. For example, in some circumstances neurological user intentiondata corresponding to a physical gesture is generated when a user thinksabout and/or focuses on a movement in the same way an amputee mightthink about moving an amputated limb.

In some embodiments, a gesture is performed without the accompaniment ofany additional input hardware. Such gestures include, for example,finger pinch movements, finger or hand swipe movements, pointinggestures, head movements (e.g., tilting, turning, or nodding of thehead), limb movements (e.g., raising or lowering of an arm or leg,flexion or extension of a joint, rotating of the hand or foot), digitmovements (e.g., finger and/or toe movements), facial movements (e.g.,smiling, furrowing the brow, intentionally blinking), full bodymovements (e.g., squatting, twisting the torso, bending at the waist,jumping), combinations of the foregoing, or other movements.

In alternative embodiments, a gesture is performed in conjunction withone or more input hardware components. Such gestures include, forexample, keyboard strokes, touch screen or touchpad swipes or taps,mouse controller clicks, click and drag movements, scroll wheel turningmovements, button pushing on a game controller, rotating or tilting anaccelerometer, and/or other gestures performed using one or more piecesof input hardware.

In some instances, a gesture is limited to a single stand-alone action.In other embodiments, a gesture includes a combination of a plurality ofone or more of the gestures described above. In such instances, acombination gesture will often, but not necessarily, correspond to acorresponding set of a plurality of related functions (e.g., a select,move and merge combination). In other embodiments, the combinationgesture will only correspond to a single function.

Neurological signals used to generate neurological user input data maybe gathered using EEG. Other embodiments utilize neurological datagathered through other means, in addition to or alternative to EEG, suchas magnetoencephalography (MEG), functional magnetic resonance imaging(fMRI), or other techniques for gathering context-based neurologicaldata. In presently preferred embodiments, non-invasive EEG techniquesare also used. It will be appreciated, however, that the scope of thisdisclosure also covers embodiments in which the described/claimed EEGsensor is replaced and/or supplemented with the MEG, fMRI and/or othercontext-based neurological data.

In this description and in the claims, the term “computing system” or“computer architecture” is defined broadly as including any standaloneor distributed device(s) and/or system(s) that include at least onephysical and tangible processor, and a physical and tangible memorycapable of having thereon computer-executable instructions that may beexecuted by the processor(s).

FIG. 1 illustrates an exemplary computer system 100 configured forchanging the state of an application in response to receivedneurological user intention data. As shown, the illustrated computersystem 100 includes memory 102 and at least one processor 104. It willbe appreciated that although only a single processor 104 is shown, thesystem 100 may also include a plurality of processors. The memory 102may be physical system memory, which may be volatile, non-volatile, orsome combination of the two. The term “memory” may also be used hereinto refer to non-volatile mass storage such as physical storage media.

The illustrated embodiment includes an EEG sensor 120 through which auser provides neurological input to the computer system 100. Thecomputer system 100 optionally includes other input/output hardware 106,including one or more keyboards, mouse controls, touch screens,microphones, speakers, display screens, track balls, scroll wheels, andthe like to enable the receiving of information from a user and fordisplaying or otherwise communicating information to a user.

The illustrated computer system 100 also includes an application 130capable of operation on the computer system 100. As shown, theapplication 130 includes a number of different application states 132 athrough 132 n (referred to generically as states 132), and a number ofapplication operations 134 a through 134 n (referred to generically asoperations 134). One or more of the application operations 134 areassociated with a specific application state or set of applicationstates 132, such that the one or more application operations 134 areonly operable when the application is in an associated applicationstate.

The application 130 may be 3D modeling software or other modelingsoftware, a video game, a virtual reality or augmented realitysimulator, an audiovisual service, a word processor, a spreadsheetapplication, a web browser, a database manager, an application forcontrolling mechanical tools and/or other machinery (e.g., for movingand operating robotic arms or other machinery), or any other applicationhaving a plurality of different application states and being capable ofperforming different operations within the different states.

Operations may include object move operations, build operations, editoperations, animation or actuation operations, data input or outputoperations, display operations, audio operations, character/objectmovement or control operations, menu operations, navigation operations,processing operations, or other operations performable within theapplication 130 to modulate input or output, settings, responses (e.g.,visual, audio, and/or mechanical responses), and the like. Although asingle exemplary application 130 is shown in this embodiment, it will beunderstood that a plurality of different applications may also beincluded with and/or added to the computer system 100.

The computer system 100 includes executable modules or executablecomponents 112 and 114. As used herein, the term “executable module” or“executable component” can refer to software objects, routings, ormethods that may be executed on the computing system. The differentcomponents, modules, engines, and services described herein may beimplemented as objects or processes that execute on the computing system(e.g., as separate threads).

The illustrated computer system 100 includes communication channels 108that enable the computing system 100 to communicate with one or moreseparate computer systems. For example, the computer system 100 may be apart of network 110, which may be configured as a Local Area Network(“LAN”), a Wide Area Network (“WAN”), or the Internet, for example. Insome embodiments, the computer system 100 communicates with and/or ispart of a distributed computer environment 150, as indicated by theplurality of separate computer systems 150 a through 150 n, each ofwhich may contain one or more of the disclosed components that are shownin system 100, entirely or partially, such as one or more of the memorycomponents, application components, or any of the other components.

The various components illustrated in FIG. 1 represent only a fewexample implementations of an integrated computer system configured forgenerating a state machine to enable changing of an application stateaccording to received user intention data. Other embodiments may dividethe described memory/storage, modules, components, and/or functionsdifferently among additional computer systems. For example, in someembodiments the training module 112 and the state machine 114 areincluded in different computer systems and/or locations, whereas inother embodiments they are included in the same standalone computersystem, as illustrated.

In some embodiments, memory components and/or program modules aredistributed across a plurality of constituent computer systems in adistributed environment. In other embodiments, memory components andprogram modules are included in a single integrated computer system.Accordingly, the systems and methods described herein are not intendedto be limited based on the particular location at which the describedcomponents are located and/or at which their functions are performed.

According to the illustrated embodiment, the memory 102 is used forstoring data and records such as the illustrated neurological userintention data 102 a, gesture library 102 b, application operationslibrary 102 c, application state library 102 d, and training profiles102 e. As explained in more detail below, one or more embodiments of thepresent disclosure are configured to receive neurological user intentiondata 102 a (e.g., as received from sensor 120), and to associate theuser intention data 102 a with corresponding application operations 102c and the corresponding gestures 102 b used to actuate the applicationoperations 102 c.

In many instances, a single gesture will be related to a plurality ofapplication operations differentiated according to the application stateat the time the gesture is performed. Accordingly, at least someembodiments associate application operations 102 c with applicationstates 102 d. In some instances, the various associations made by usingthe embodiments described herein are stored in the training profiles 102e, as shown. In other embodiments, the training profiles 102 e and/orother data elements shown in FIG. 1 are stored at one or more locationsremote to the computer system 100. For example, the training profiles102 e may be stored at a third party server (e.g., a game server) thataccesses and applies the training profiles 102 e, when necessary.

In some embodiments, the training profiles 102 e are broken downaccording to individual users, groups of individual users (e.g., afamily profile), individual applications, and/or sets of applications inorder to, for example, enable formation of a more fine-tuned profilebased on individual differences in neurological signal generation.

The illustrated computer system 100 includes a training module 112configured to generate, update, and/or manage a state machine 114. Thestate machine 114 is configured to receive neurological user intentiondata from the sensor 120, and to translate the user intention data intoapplication instructions for changing an application state of anapplication 130 according to the received neurological user intentiondata.

The training module 112 is configured to detect neurological userintention data and associate the user intent data to a correspondingapplication operation. For example, in some implementations the trainingmodule 112 operates in conjunction with a training application ortraining wizard to provide a series of instructions and/or prompts toguide a user through a training process for a particular application orset of applications. In one example, a prompt directs a user to performa particular application operation while the training module 112monitors the corresponding neurological data generated by the user.Alternatively, or additionally, the training module 112 runs in thebackground as the user runs the application 130, functioning to monitorapplication activity and the corresponding neurological signalsgenerated by the user during normal use of the application 130. Duringuse, when the user performs a gesture to actuate a particularcorresponding operation, the neurological signature generated by theuser is associated with the particular application operation.

Although a particular gesture may actuate a plurality of differentapplication operations, as described above, the generated neurologicaluser intent data associated with at least some of the differentoperations includes detectable differences based on the fact that theuser intends a different effect when performing the particular gesture,according to the particular application state the user is aware of atthe time of the gesture. The generated user intent data corresponding tothe particular application operation thereby provides a means fordistinguishing between different application operations, even though thedifferent operations rely upon the same underlying gesture foractuation. By associating the neurological user intent data to theapplication operations, and by associating/mapping the applicationoperations to the application states within which they are operable, theneurological user intent data is usable to align the application statewith the user's intent, in order to perform the correct and desiredapplication operation.

In some embodiments, the training module 112 is also configured to parsethe application 130 to determine various application operations 134 thatrely on a shared underlying gesture, and to associate these variousapplication operations to the different application states 132 withinwhich they are operable. Additionally, or alternatively, the gestures,operations, and/or states relevant to a particular training process ortraining application may be user-selected according to user needs andpreferences.

In some embodiments, the training module 112 is configured to map orotherwise associate received neurological data to correspondingapplication operations by generating one or more model(s) or other datastructure(s) that correlate/map the neurological data to thecorresponding intended application operations. In some embodiments, thetraining module 112 includes signal processing functionality to providefiltering (e.g., low and/or high band-pass filtering and/or filtering ofdelta and gamma EEG waves), artifact removal (e.g., removal of commonEEG artifacts known to be associated with blinks, yawns, extraneousaudio or visual stimulation, or other data and movements that are notcorrelated to the gesture underlying the intended applicationoperation), and/or other processing of one or more signals of thereceived neurological data 102 a.

The mapping may include averaging, the application of best-fitalgorithms, or any other statistical or correlation mappingalgorithm(s). In some embodiments, the training module 112 is operableto perform regression analysis on the received neurological data. Forexample, different user intentions for a particular gesture maycorrelate to different percentages of the power spectrum (e.g., asdetermined through Fourier analysis) within the different wave bands(alpha, beta, gamma, delta) of the corresponding EEG signals, maycorrelate to an amount of phase and/or magnitude synchrony, and/or maycorrelate to other characteristics of the corresponding EEG signals.

In preferred embodiments, the training module 112 is configured to usemachine learning techniques to correlate the received EEG signalinformation to the corresponding intended application operations inorder to generate a predictive model that can provide the intendedapplication operation as output based on a neurological signal input.

The illustrated computer system 100 also includes a state machine 114configured to apply one or more correlations/associations (e.g., asstored in training profiles 102 e) to received neurological userintention data to generate application instructions for controlling thestate of the application 130, according to the predicted intent of theuser. In some circumstances, a training process as described herein isused to create or update a state machine by associating applicationoperations 134 and application states 132 with neurological userintention data 102 a. The state machine 114 can subsequently receive aneurological user intention data input from the sensor 120 and can usethe input to determine the user intended operation corresponding to theinput (e.g., according to one or more regression models and/or machinelearning models generated during a training process and applied by thestate machine).

In some embodiments, the state machine 114 is operable even in theabsence of a prior training process. For example, some applications havesufficient similarity to applications for which an associated statemachine has already been created, to enable use of the state machinewith an application without the need for a specific training processand/or generation of a new state machine prior to use with theapplication.

Additionally, or alternatively, creating a state machine includesobtaining neurological data from a plurality of different users andbuilding a database of obtained neurological data from a plurality ofusers. For example, in some embodiments the computer system 100 enablesdownloading (e.g., through network 110) of a generic state machine for aparticular application from a third party or a state machine service. Ageneric state machine includes, for example, predictive models generatedthrough measured EEG intent data for a plurality of users while theusers perform context-specific operations. In some embodiments, EEGintent data or other neurological user intent data usable for creatingand/or updating a state machine includes data from a database alreadycontaining stored neurological data (e.g., from a plurality of users).

In some embodiments, a generic state machine provides immediate “plugand play” functionality, allowing the computer system to forego aninitial training process prior to use of the state machine with anapplication. Optionally, the computer system 100 enables a user tomodify or fine-tune a generic state machine to generate anindividualized or customized state machine tailored to an individual orgroup of individuals. In addition, some embodiments enable creation of astate machine from scratch, without the use of or modification of ageneric state machine.

In preferred embodiments, the state machine 114 is configured to changethe state of an application to enable execution of the intendedoperation with minimal latency. In at least some circumstances, thestate machine 114 operates to change the application state substantiallyin real-time, so that from the perspective of the user, the applicationperforms the state change without any additional latency beyond thelatency inherent in processing and applying the state change in theparticular application.

In the description that follows, embodiments are described withreference to acts that are performed by one or more computing systems.If such acts are implemented in software, one or more processors of theassociated computing system that performs the act direct the operationof the computing system in response to the processor(s) of the computingsystem having executed computer-executable instructions that areembodied on one or more computer-readable media (e.g., hardware storagedevice(s)). An example of such an operation involves the manipulation ofdata.

The computer-executable instructions (and the manipulated data) may bestored in the memory 102 of the computer system 100, and/or in one ormore separate computer system components. The computer-executableinstructions may be used to implement and/or instantiate all of thefunctionality disclosed herein, including the functionality that isdisclosed in reference to one or more of the flow diagrams of FIGS. 2through 4.

FIG. 2 is a flowchart 200 of a computer-implemented method for usingneurological user intention data to change the state of an application.As shown, a computer system operates an application that is configuredto change state and that is further configured to perform a firstoperation in response to detecting a particular gesture while operatingin a first state and to perform a second operation in response todetecting the particular gesture while operating in a second state (act210). By way of example, the application may be a word processingapplication, and the operation may be a paste operation following thecopying of a selection of text. In a first state, the application maypaste the selection with formatting that matches the destination,whereas in a second state, the application may paste the selection withformatting that matches the source, for example.

The computer system subsequently detects a user input while theapplication is operating in the first state, the detected user inputcomprising neurological user intention data that is associated with theparticular gesture in the context of the second operation (act 220).Continuing with the word processing application example, the wordprocessing application may be set to paste the selection withdestination formatting, while the computer system detects neurologicaluser intention data associated with a paste command gesture (e.g., aCtrl+v keyboard stroke) as associated with an intent to paste theselection with source formatting.

The computing system then changes the application from the first stateto the second state in response to detecting the user input comprisingthe neurological intention data (act 230). For example, the wordprocessing application is changed from the destination-formatting stateto the source-formatting state so as to align with the user's intentionin performing the paste operation, as indicated by the receivedneurological user intention data.

FIG. 3 is a flowchart 300 of a computer-implemented method for creatingand/or managing a state machine for at least one application. As shown,a computer system associates a set of application operations to acorresponding set of application states, wherein a particular gesturecauses the application to perform a particular operation of the set ofoperations based on a corresponding application state (act 310). Asdescribed by the foregoing, and as shown by the illustrated examplesdescribed in detail below, a particular gesture can cause the executionof different operations in an application depending on the particularstate of the application.

The computer system also detects first neurological user intention datagenerated by a user during performance of the particular gesture that isoperative to cause the application to perform a first operation when theapplication is in the first state (act 320), and detects secondneurological user intention data generated by a user during performanceof the particular gesture that is operative to cause the application toperform a second operation when the application is in the second state(act 330). In an example of an augmented reality gaming application, thecomputer system monitors and detects user intention data while a user isperforming an “air tap” gesture to select an object (e.g., a hologramobject) when the gaming application is in a selection state, andmonitors and detects user intention data while a user is performing thesame air tap gesture to fire a virtual projectile when the applicationis in an action/play state.

The detection of the first neurological user intention data may beperformed in conjunction with a training application or wizard and/ormay include the background monitoring of user intention data while theapplication is put to normal use. Continuing the gaming applicationexample, a training exercise may be utilized to collect sufficientneurological data associated with performance of the air tap gesture inthe different contexts of the object selection operation and the virtualprojectile operation so as to be able to build or update the statemachine to distinguish the different neurological signatures of the userrespectively associated with the different contexts. Additionally, oralternatively, a training program may run in the background while theuser plays the game to gather neurological data to build and/or updatethe state machine.

The computer system subsequently associates the first neurological userintention data with the first operation in the first state and thesecond neurological user intention data with the second operation in thesecond state (act 340). For example, the neurological data associatedwith the air tap gesture performed with the intention of selecting anobject is associated with the select object operation, and theneurological data associated with the air tap gesture performed with theintention of firing a projectile is associated with the fire projectileoperation. Accordingly, the resulting generated state machine enablesthe user to perform the air tap gesture to perform the intended objectselection or projectile firing operation without the need of manuallyswitching between a selection state and a firing state, as therespectively associated neurological user intent data enables theapplication to automatically change to the appropriate state to enableexecution of the intended operation.

FIG. 4 is a flowchart 400 of an alternative computer-implemented methodfor creating and/or managing a state machine for at least oneapplication. As shown, a computer system associates a set of applicationoperations to a corresponding set of application states, wherein aparticular gesture causes the application to perform a particularoperation of the set of operations based on a corresponding applicationstate (act 410).

The computer system also detects first neurological user intention datagenerated by a user during a user initiated change of application stateto a particular application state, wherein the particular applicationstate is required by the application to perform the particular operationin response to the particular gesture (act 420), and detects secondneurological user intention data generated by a user during performanceof the particular gesture that is operative to cause the application toperform the particular operation (act 430). Using an alternative versionof the augmented reality gaming application example, the computer systemmonitors and detects user intention data while a user manually changesthe game from an object selection state to a projectile firing state,and monitors and detects user intention data after the user has changedthe game state and is performing the air tap gesture to fire a virtualprojectile.

The computer system subsequently associates the first neurological userintention data with the particular application state and the secondneurological user intention data with the particular operation (act440). For example, the neurological data associated with intentionallychanging the game state to the projectile firing mode is associated withthe projectile firing game state, and the neurological data associatedwith the air tap gesture performed with the intention of firing aprojectile is associated with the fire projectile operation.

Accordingly, the resulting generated state machine enables the user tocontrol state changing activity of the application, and correspondingcontext-based operations within the selected application state, usingneurological user intent data. In some embodiments, for example, thegenerated user intent data operates to change the application to thedesired state as a user thinks about and/or focuses on making the statechange, thereby generating the actuating neurological intent data, asopposed to manually performing the gesture that causes the state changeto occur.

The following examples illustrate operation and functionality of variousexemplary embodiments for changing the state of an application usingneurological user intent data. The scope of the concepts and featuresdescribed herein and recited in the claims is not limited to theseparticular illustrated examples, nor the referenced types ofapplications. Other embodiments include different configurations andcombinations of applications, application states, applicationoperations, and/or associated neurological user intent data. Someadditional non-limiting examples will now be provided to further clarifythis point.

FIGS. 5A-5D illustrate operation of a state machine configured to changethe state of a 3D modeling application using neurological user intentdata, as an implicit user request that is needed to perform a desiredfunction based on a particular gesture. FIGS. 5A-5D illustrate a userinterface 500 showing a 3D object 510 within a canvas 520. The interface500 also includes a pointer 550, a label 530 displaying the currentapplication state and a dropdown menu 540 that enables manual changingof the application state. The 3D application is controllable through amouse controller, a touch screen, a touchpad, and/or other input device,for example. In an alternative example, the 3D modeling application isoperable in a virtual reality or augmented reality environment, wherethe application is controllable through user hand and finger gestures.Although a pointer 550 is illustrated in this example, it should beunderstood that other embodiments, particularly virtual reality oraugmented reality applications and touch screen applications, thepointer 550 may be omitted and/or be controlled with a hand or otherobject.

When the application is in “state A,” as shown in FIG. 5A, a click anddrag gesture or a tap and drag gesture executes a move operation thatmoves the 3D object 510 in the direction of the drag motion, asillustrated. When the application is in “state B,” as shown in FIG. 5B,the same click and drag gesture executes a rotate operation that rotatesthe 3D object 510, as illustrated. When the application is in “state C,”as shown in FIG. 5C, the same click and drag gesture executes an objectextend operation that extends a selected face or point of the 3D object510, as illustrated.

FIG. 5D illustrates operation of the state machine to automaticallychange the state of the 3D modeling application. As shown, a user 560 isfitted with a sensor 570 configured to sense and transmit neurologicaluser intent data during interaction with the interface 500. Before aclick and drag action is performed, the application is in state A, whichassociates the click and drag action with a move operation. The user,however, intends to perform an extend operation (as illustrated bythought bubble 580). As the user performs the click and drag action, thestate machine receives neurological user intention data corresponding tothe intended extend operation. In response, the state machine operatesto change the application to state C to provide execution of theintended/desired extend operation, without requiring the user tomanually or explicitly (through a menu or separate input) change theapplication state.

FIG. 6 illustrates an alternative mode of operation of a state machineto change the state of an application. In this example, the user doesexplicitly change application state, but does so with neurological userintent data rather than manual controls. Continuing with an exemplary 3Dmodeling application, a user 660 is fitted with a sensor 670 configuredto sense and transmit neurological user intent data during interactionwith user interface 600. The interface 600 includes a 3D object 610within a canvas, a label 630 displaying the current application state,and a dropdown menu 640 enabling manual changing of the state upon userselection.

Initially, the application is in state A, which associates a click anddrag action with a move operation. The user 660, however, intends toperform an extend operation, which requires the application to be instate C. Before performing the click and drag operation, the user 660 isaware that the application state needs to be changed to state C andintends to change the application to state C (as illustrated by thoughtbubble 680). As a result, the user generates neurological user intentiondata corresponding to a state change operation for changing theapplication to state C. In response, the state machine operates tochange the application state to state C (using menu 690), after whichthe user 660 performs the click and drag gesture to execute the extendoperation.

It will be appreciated, from the foregoing description, that varioustypes of applications can be configured to utilize neurological userintent data to control state changes (explicitly) or automatically as animplicit part of performing a function based on a gesture and the userintent. Such embodiments can be used to provide increased userconvenience and computational efficiencies over existing systems.

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer system that includes computer hardware, suchas, for example, one or more processors and system memory. Embodimentswithin the scope of the present invention also include physical andother computer-readable media for carrying or storingcomputer-executable instructions and/or data structures. Suchcomputer-readable media can be any available media that can be accessedby a general-purpose or special-purpose computer system.Computer-readable media that store computer-executable instructionsand/or data structures are computer storage media. Computer-readablemedia that carry computer-executable instructions and/or data structuresare transmission media. Thus, by way of example, and not limitation,embodiments of the invention can comprise at least two distinctlydifferent kinds of computer-readable media: computer storage media andtransmission media.

Computer storage media are physical storage media that storecomputer-executable instructions and/or data structures. Physicalstorage media include computer hardware, such as RAM, ROM, EEPROM, solidstate drives (“SSDs”), flash memory, phase-change memory (“PCM”),optical disk storage, magnetic disk storage or other magnetic storagedevices, or any other hardware storage device(s) which can be used tostore program code in the form of computer-executable instructions ordata structures, which can be accessed and executed by a general-purposeor special-purpose computer system to implement the disclosedfunctionality of the invention.

Transmission media can include a network and/or data links which can beused to carry program code in the form of computer-executableinstructions or data structures, and which can be accessed by ageneral-purpose or special-purpose computer system. A “network” isdefined as one or more data links that enable the transport ofelectronic data between computer systems and/or modules and/or otherelectronic devices. When information is transferred or provided over anetwork or another communications connection (either hardwired,wireless, or a combination of hardwired or wireless) to a computersystem, the computer system may view the connection as transmissionmedia. Combinations of the above should also be included within thescope of computer-readable media.

Further, upon reaching various computer system components, program codein the form of computer-executable instructions or data structures canbe transferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a “NIC”), and theneventually transferred to computer system RAM and/or to less volatilecomputer storage media at a computer system. Thus, it should beunderstood that computer storage media can be included in computersystem components that also (or even primarily) utilize transmissionmedia.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at one or more processors, cause ageneral-purpose computer system, special-purpose computer system, orspecial-purpose processing device to perform a certain function or groupof functions. Computer-executable instructions may be, for example,binaries, intermediate format instructions such as assembly language, oreven source code.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, virtual or augmented realityheadsets, and the like. The invention may also be practiced indistributed system environments where local and remote computer systems,which are linked (either by hardwired data links, wireless data links,or by a combination of hardwired and wireless data links) through anetwork, both perform tasks. As such, in a distributed systemenvironment, a computer system may include a plurality of constituentcomputer systems. In a distributed system environment, program modulesmay be located in both local and remote memory storage devices.

Those skilled in the art will also appreciate that the invention may bepracticed in a cloud computing environment. Cloud computing environmentsmay be distributed, although this is not required. When distributed,cloud computing environments may be distributed internationally withinan organization and/or have components possessed across multipleorganizations. In this description and the following claims, “cloudcomputing” is defined as a model for enabling on-demand network accessto a shared pool of configurable computing resources (e.g., networks,servers, storage, applications, and services). The definition of “cloudcomputing” is not limited to any of the other numerous advantages thatcan be obtained from such a model when properly deployed.

A cloud computing model can be composed of various characteristics, suchas on-demand self-service, broad network access, resource pooling, rapidelasticity, measured service, and so forth. A cloud computing model mayalso come in the form of various service models such as, for example,Software as a Service (“SaaS”), Platform as a Service (“PaaS”), andInfrastructure as a Service (“IaaS”). The cloud computing model may alsobe deployed using different deployment models such as private cloud,community cloud, public cloud, hybrid cloud, and so forth.

Some embodiments, such as a cloud computing environment, may comprise asystem that includes one or more hosts that are each capable of runningone or more virtual machines. During operation, virtual machines emulatean operational computing system, supporting an operating system andperhaps one or more other applications as well. In some embodiments,each host includes a hypervisor that emulates virtual resources for thevirtual machines using physical resources that are abstracted from viewof the virtual machines. The hypervisor also provides proper isolationbetween the virtual machines. Thus, from the perspective of any givenvirtual machine, the hypervisor provides the illusion that the virtualmachine is interfacing with a physical resource, even though the virtualmachine only interfaces with the appearance (e.g., a virtual resource)of a physical resource. Examples of physical resources includingprocessing capacity, memory, disk space, network bandwidth, mediadrives, and so forth.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A computer-implemented method for changing astate of an application using neurological data, the method beingimplemented by a computing system that includes at least one processorand one or more hardware storage devices having storedcomputer-executable instructions that are executable by the at least oneprocessor to cause the computing system to perform at least thefollowing: operating an application that includes at least a firstapplication state, a second application state, and an application statedependent operation, wherein invoking the application state dependentoperation in response to detecting a particular user input causes afirst action within the application while the application is operatingin the first application state but causes a second action within theapplication while the application is operating in the second applicationstate; while the application is operating in the first applicationstate, detecting the particular user input invoking the applicationstate dependent operation, the particular user input includingneurological user intention data indicating that the user intends theparticular input to cause the second action in accordance with thesecond application state; based on detecting the particular user input,automatically changing the application from the first application stateto the second application state; and causing the second action to beexecuted at the application.
 2. The method of claim 1, furthercomprising performing the second operation in response to detecting theuser input comprising the neurological user intention data.
 3. Themethod of claim 1, wherein the application operates in a virtual realityor an augmented reality environment.
 4. The method of claim 1, whereinthe neurological user intention data of the detected user input includeselectroencephalography readings that are generated from user performanceof the particular gesture with an intention of causing the secondoperation through performance of the particular gesture.
 5. The methodof claim 1, further comprising: detecting a subsequent user input whilethe application is operating in the second application state, afterbeing changed from the first application state to the second applicationstate, the detected subsequent user input comprising neurological userintention data that is associated with the particular gesture accordingto the first operation; and changing the application from the secondapplication state to the first application state in response todetecting the subsequent user input.
 6. The method of claim 5, furthercomprising performing the first operation in response to detecting thesubsequent user input.
 7. The method of claim 5, wherein theneurological user intention data of the detected subsequent user inputincludes electroencephalography readings that are generated from userperformance of the particular gesture with an intention of causing thefirst operation through performance of the particular gesture.
 8. Themethod of claim 1, wherein the particular gesture is performed throughone or more user movements without the accompaniment of additional inputhardware.
 9. The method of claim 1, wherein the neurological userintention data corresponding to the particular gesture is generatedwithout physical performance of the particular gesture.
 10. The methodof claim 1, wherein the particular gesture is performed using one ormore input hardware components.
 11. The method of claim 1, wherein theapplication is a modeling application.
 12. A computer-implemented methodfor creating and managing a state machine for at least one application,the state machine being configured to change a state of the at least oneapplication based on neurological data received from a user, the methodbeing implemented by a computing system that includes at least oneprocessor and one or more hardware storage devices having stored thereoncomputer-executable instructions that are executable by the at least oneprocessor to cause the computing system to perform at least thefollowing: associating a set of application state dependent operationsto a corresponding set of application states within an applicationconfigured to include multiple application states, a particular userinput causing the application to perform a particular application statedependent operation from the set of application state dependentoperations based on a corresponding application state, wherein theapplication includes at least a first application state associated witha first application state dependent operation and a second applicationstate associated with a second application state dependent operation; asa result of detecting a first neurological user intention data generatedby a user during performance of the particular user input, causing theapplication to perform a first application state dependent operationwhen the application is in the first application state; as a result ofdetecting a second neurological user intention data generated by a userduring performance of the particular user input, causing the applicationto perform a second application state dependent operation when theapplication is in the second application state; and as a result ofdetecting the second neurological user intention data generated by auser during performance of the particular user input while theapplication is operating in the first application state, automaticallychanging the application from the first application state to the secondapplication state and causing the second application state operation tobe executed at the application.
 13. The method of claim 12, wherein thefirst and second neurological user intention data are generated by asingle user, and wherein the state machine is thereby tailored to thesingle user.
 14. The method of claim 12, wherein the first neurologicaluser intention data with the first operation in the first state and thesecond neurological user intention data with the second operation in thesecond state using machine learning techniques.
 15. The method of claim12, wherein the first and second neurological user intention datainclude electroencephalography readings that are generated from userperformance of the particular gesture with an intention of causing thefirst and second operation, respectively.
 16. A computer-implementedmethod for creating and managing a state machine for at least oneapplication, the state machine being configured to change a state of theat least one application based on neurological data received from auser, the method being implemented by a computing system that includesat least one processor and one or more hardware storage devices havingstored thereon computer-executable instructions that are executable bythe at least one processor to cause the computing system to perform atleast the following: operating an application that is configured toinclude at least a first application state, a second application state,and an application state dependent operation, wherein invoking theapplication state dependent operation in response to detecting aparticular user input causes a first action within the application whilethe application is operating in the first application state but causes asecond action within the application while the application is operatingin the second application state; while the application is operating inthe first application state, detecting the particular user inputinvoking the application state dependent operation, the particular userinput including neurological user intention data indicating that theuser intends the particular input to cause the second action inaccordance with the second application state; based on detecting theparticular user input, automatically changing the application from thefirst application state to the second application state; and causing thesecond action to be executed at the application.